Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Test Package Versions Bump] Updating package versions #4792

Merged
merged 3 commits into from
Nov 2, 2023
Merged

[Test Package Versions Bump] Updating package versions #4792

merged 3 commits into from
Nov 2, 2023

Conversation

github-actions[bot]
Copy link
Contributor

Updates the package versions for integration tests.

@github-actions github-actions bot requested review from a team as code owners October 31, 2023 17:06
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Oct 31, 2023 •
edited
Loading

Datadog Report

Branch report: bot/test-package-versions-bump
Commit report: 6f1e320

❄️ dd-trace-dotnet: 0 Failed, 1 New Flaky, 297495 Passed, 942 Skipped, 37m 11.59s Wall Time

New Flaky Tests (1)

  • TransportsWorkCorrectly - Datadog.Trace.Tests.DataStreamsMonitoring.DataStreamsMonitoringTransportTests - Last Failure

    Expand for error 
     Expected varpayload=result to contain a single item, but found {
     Datadog.Trace.TestHelpers.DataStreamsMonitoring.MockDataStreamsPayload
     {
         Env = "env", 
         Lang = "dotnet", 
         PrimaryTag = <null>, 
         Service = "service", 
         Stats = {
             Datadog.Trace.TestHelpers.DataStreamsMonitoring.MockDataStreamsBucket
             {
 ...

@andrewlock
Copy link
Member

andrewlock commented Oct 31, 2023 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4792) - mean (71ms)  : 63, 79
     .   : milestone, 71,
    master - mean (71ms)  : 62, 79
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4792) - mean (996ms)  : 979, 1013
     .   : milestone, 996,
    master - mean (992ms)  : 972, 1012
     .   : milestone, 992,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4792) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (106ms)  : 102, 109
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4792) - mean (686ms)  : 661, 711
     .   : milestone, 686,
    master - mean (680ms)  : 666, 694
     .   : milestone, 680,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4792) - mean (90ms)  : 88, 93
     .   : milestone, 90,
    master - mean (89ms)  : 86, 92
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4792) - mean (656ms)  : 632, 680
     .   : milestone, 656,
    master - mean (652ms)  : 631, 672
     .   : milestone, 652,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4792) - mean (188ms)  : 185, 192
     .   : milestone, 188,
    master - mean (188ms)  : 185, 190
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4792) - mean (1,140ms)  : 1119, 1160
     .   : milestone, 1140,
    master - mean (1,140ms)  : 1120, 1161
     .   : milestone, 1140,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4792) - mean (271ms)  : 267, 275
     .   : milestone, 271,
    master - mean (271ms)  : 268, 274
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4792) - mean (1,097ms)  : 1071, 1123
     .   : milestone, 1097,
    master - mean (1,094ms)  : 1068, 1121
     .   : milestone, 1094,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4792) - mean (262ms)  : 258, 266
     .   : milestone, 262,
    master - mean (262ms)  : 259, 265
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4792) - mean (1,070ms)  : 1043, 1097
     .   : milestone, 1070,
    master - mean (1,069ms)  : 1038, 1101
     .   : milestone, 1069,
Loading

@andrewlock
Copy link
Member

andrewlock commented Oct 31, 2023 •
edited
Loading

Benchmarks Report 🐌

Benchmarks for #4792 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.333
  • 3 benchmarks are slower, with geometric mean 1.158
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8μs 41.1ns 193ns 0.0201 0.00803 0 7.29 KB
master StartStopWithChild netcoreapp3.1 9.87μs 54ns 319ns 0.0194 0.00485 0 7.39 KB
master StartStopWithChild net472 15.7μs 65.2ns 253ns 1.28 0.316 0.108 7.67 KB
#4792 StartStopWithChild net6.0 7.96μs 40.6ns 233ns 0.0232 0.0116 0 7.29 KB
#4792 StartStopWithChild netcoreapp3.1 9.8μs 52.3ns 281ns 0.024 0.00958 0 7.38 KB
#4792 StartStopWithChild net472 15.8μs 51.9ns 194ns 1.29 0.332 0.108 7.67 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 468μs 120ns 431ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 633μs 122ns 472ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 800μs 272ns 1.05μs 0.401 0 0 3.3 KB
#4792 WriteAndFlushEnrichedTraces net6.0 467μs 188ns 702ns 0 0 0 2.7 KB
#4792 WriteAndFlushEnrichedTraces netcoreapp3.1 626μs 277ns 1.04μs 0 0 0 2.7 KB
#4792 WriteAndFlushEnrichedTraces net472 807μs 361ns 1.3μs 0.403 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4792

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.427 173.53 121.58

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 43.1μs 93.3ns 361ns 0.0218 0 0 2.03 KB
master AllCycleSimpleBody netcoreapp3.1 45.3μs 70.6ns 274ns 0.0227 0 0 2.01 KB
master AllCycleSimpleBody net472 46.7μs 80ns 310ns 0.322 0 0 2.08 KB
master AllCycleMoreComplexBody net6.0 229μs 269ns 971ns 0.114 0 0 8.63 KB
master AllCycleMoreComplexBody netcoreapp3.1 233μs 126ns 487ns 0.117 0 0 8.52 KB
master AllCycleMoreComplexBody net472 239μs 53.7ns 201ns 1.31 0 0 8.7 KB
master ObjectExtractorSimpleBody net6.0 174ns 0.0789ns 0.305ns 0.00396 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 174ns 0.143ns 0.553ns 0.00375 0 0 272 B
master ObjectExtractorSimpleBody net472 145ns 0.135ns 0.505ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.1μs 1.24ns 4.66ns 0.0551 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.07μs 1.73ns 6.25ns 0.0528 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.15μs 3.8ns 14.7ns 0.617 0.00623 0 3.89 KB
#4792 AllCycleSimpleBody net6.0 43.6μs 88.3ns 342ns 0.022 0 0 2.03 KB
#4792 AllCycleSimpleBody netcoreapp3.1 44.6μs 157ns 606ns 0.0224 0 0 2.01 KB
#4792 AllCycleSimpleBody net472 47.1μs 26.9ns 104ns 0.325 0 0 2.08 KB
#4792 AllCycleMoreComplexBody net6.0 232μs 156ns 604ns 0.114 0 0 8.63 KB
#4792 AllCycleMoreComplexBody netcoreapp3.1 232μs 86.8ns 313ns 0 0 0 8.52 KB
#4792 AllCycleMoreComplexBody net472 236μs 112ns 435ns 1.29 0 0 8.7 KB
#4792 ObjectExtractorSimpleBody net6.0 122ns 0.0973ns 0.377ns 0.00394 0 0 280 B
#4792 ObjectExtractorSimpleBody netcoreapp3.1 176ns 0.097ns 0.363ns 0.00374 0 0 272 B
#4792 ObjectExtractorSimpleBody net472 151ns 0.188ns 0.728ns 0.0446 0 0 281 B
#4792 ObjectExtractorMoreComplexBody net6.0 3.05μs 1.47ns 5.51ns 0.0537 0 0 3.88 KB
#4792 ObjectExtractorMoreComplexBody netcoreapp3.1 4.12μs 2.11ns 8.18ns 0.0517 0 0 3.78 KB
#4792 ObjectExtractorMoreComplexBody net472 4.16μs 2.42ns 9.05ns 0.617 0.00621 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 12.7μs 2.92ns 10.9ns 0.134 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 19.3μs 3.39ns 12.7ns 0.126 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) net472 28.5μs 21.5ns 83.2ns 1.49 0 0 9.48 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 74.7μs 35.3ns 137ns 0.188 0 0 15.77 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 85.3μs 167ns 579ns 0.214 0 0 15.72 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 96μs 69.9ns 252ns 2.54 0 0 16.04 KB
master RunWaf(args=NestedMap (100)) net6.0 24μs 10.6ns 39.6ns 0.275 0 0 19.66 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 38.2μs 13ns 50.4ns 0.266 0 0 20.42 KB
master RunWaf(args=NestedMap (100)) net472 54.5μs 17.9ns 69.2ns 3.27 0.0273 0 20.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 91.7μs 26.4ns 102ns 0.364 0 0 26.01 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 108μs 59.8ns 232ns 0.328 0 0 26.72 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 132μs 108ns 420ns 4.3 0.0652 0 27.19 KB
master RunWaf(args=NestedMap (20)) net6.0 23.4μs 9.49ns 36.8ns 0.27 0 0 19.4 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 37.6μs 21.3ns 82.6ns 0.263 0 0 19.84 KB
master RunWaf(args=NestedMap (20)) net472 57.7μs 33.9ns 131ns 3.17 0.0576 0 20.04 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 91.4μs 63.1ns 244ns 0.323 0 0 25.74 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 108μs 121ns 469ns 0.323 0 0 26.15 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 131μs 150ns 583ns 4.21 0.0647 0 26.6 KB
#4792 RunWaf(args=NestedMap (10)) net6.0 12.8μs 3.97ns 14.9ns 0.135 0 0 9.42 KB
#4792 RunWaf(args=NestedMap (10)) netcoreapp3.1 19.5μs 3.91ns 14.6ns 0.127 0 0 9.42 KB
#4792 RunWaf(args=NestedMap (10)) net472 28.1μs 19.8ns 76.8ns 1.5 0 0 9.48 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 76.8μs 43.2ns 168ns 0.231 0 0 15.77 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 85.3μs 199ns 773ns 0.17 0 0 15.72 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [22]) net472 98.1μs 215ns 834ns 2.5 0 0 16.04 KB
#4792 RunWaf(args=NestedMap (100)) net6.0 23.9μs 7.08ns 26.5ns 0.276 0 0 19.66 KB
#4792 RunWaf(args=NestedMap (100)) netcoreapp3.1 38.2μs 11ns 42.8ns 0.268 0 0 20.42 KB
#4792 RunWaf(args=NestedMap (100)) net472 57.4μs 33.6ns 126ns 3.26 0.0572 0 20.63 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 93.3μs 31.2ns 121ns 0.326 0 0 26.01 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 110μs 104ns 402ns 0.331 0 0 26.72 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [23]) net472 132μs 59.2ns 229ns 4.27 0.0657 0 27.19 KB
#4792 RunWaf(args=NestedMap (20)) net6.0 23.9μs 9.02ns 35ns 0.276 0 0 19.4 KB
#4792 RunWaf(args=NestedMap (20)) netcoreapp3.1 37.7μs 14ns 50.6ns 0.265 0 0 19.84 KB
#4792 RunWaf(args=NestedMap (20)) net472 57μs 42.6ns 165ns 3.16 0.057 0 20.04 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 91.5μs 56.1ns 210ns 0.362 0 0 25.74 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 108μs 377ns 1.46μs 0.321 0 0 26.14 KB
#4792 RunWafWithAttack(args=Neste(...)tack) [22]) net472 132μs 55.8ns 216ns 4.17 0.0661 0 26.6 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 168μs 207ns 801ns 0.167 0 0 18.04 KB
master SendRequest netcoreapp3.1 186μs 160ns 598ns 0.186 0 0 20.2 KB
master SendRequest net472 0.000151ns 8.5E‑05ns 0.000318ns 0 0 0 0 b
#4792 SendRequest net6.0 169μs 169ns 653ns 0.169 0 0 18.04 KB
#4792 SendRequest netcoreapp3.1 186μs 235ns 911ns 0.185 0 0 20.2 KB
#4792 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 563μs 627ns 2.43μs 0.568 0 0 41.61 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 649μs 533ns 1.92μs 0.322 0 0 41.64 KB
master WriteAndFlushEnrichedTraces net472 833μs 3.51μs 13.1μs 8.41 2.4 0.401 53.26 KB
#4792 WriteAndFlushEnrichedTraces net6.0 544μs 436ns 1.63μs 0.548 0 0 41.54 KB
#4792 WriteAndFlushEnrichedTraces netcoreapp3.1 656μs 745ns 2.89μs 0.322 0 0 41.72 KB
#4792 WriteAndFlushEnrichedTraces net472 840μs 2.69μs 10.4μs 8.25 2.6 0.434 53.21 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.05μs 0.679ns 2.63ns 0.0105 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.35μs 0.46ns 1.72ns 0.0102 0 0 768 B
master ExecuteNonQuery net472 1.6μs 0.652ns 2.53ns 0.116 0 0 730 B
#4792 ExecuteNonQuery net6.0 1.05μs 1.62ns 5.83ns 0.0105 0 0 768 B
#4792 ExecuteNonQuery netcoreapp3.1 1.37μs 0.506ns 1.96ns 0.0102 0 0 768 B
#4792 ExecuteNonQuery net472 1.66μs 0.672ns 2.6ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4792

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.149 1,102.10 1,265.98

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.1μs 0.463ns 1.79ns 0.0128 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.27μs 0.326ns 1.22ns 0.0125 0 0 936 B
master CallElasticsearch net472 2.36μs 1.12ns 4.02ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.2μs 0.518ns 1.94ns 0.0127 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.42μs 0.697ns 2.51ns 0.0135 0 0 984 B
master CallElasticsearchAsync net472 2.41μs 1.45ns 5.63ns 0.16 0 0 1.01 KB
#4792 CallElasticsearch net6.0 1.27μs 0.838ns 2.9ns 0.0134 0 0 936 B
#4792 CallElasticsearch netcoreapp3.1 1.37μs 0.644ns 2.49ns 0.0125 0 0 936 B
#4792 CallElasticsearch net472 2.38μs 1.09ns 4.22ns 0.151 0 0 955 B
#4792 CallElasticsearchAsync net6.0 1.17μs 0.583ns 2.18ns 0.0129 0 0 912 B
#4792 CallElasticsearchAsync netcoreapp3.1 1.48μs 0.383ns 1.43ns 0.0133 0 0 984 B
#4792 CallElasticsearchAsync net472 2.45μs 0.865ns 3.35ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.2μs 1.09ns 4.24ns 0.0127 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.48μs 0.753ns 2.82ns 0.0119 0 0 912 B
master ExecuteAsync net472 1.64μs 1.06ns 4.09ns 0.138 0 0 875 B
#4792 ExecuteAsync net6.0 1.21μs 0.978ns 3.66ns 0.0128 0 0 912 B
#4792 ExecuteAsync netcoreapp3.1 1.46μs 3.01ns 11.3ns 0.0123 0 0 912 B
#4792 ExecuteAsync net472 1.67μs 1.59ns 6.16ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.71μs 1.78ns 6.66ns 0.0261 0 0 1.9 KB
master SendAsync netcoreapp3.1 4.4μs 1.89ns 7.06ns 0.033 0 0 2.43 KB
master SendAsync net472 7.14μs 1.71ns 6.17ns 0.475 0.00357 0 2.99 KB
#4792 SendAsync net6.0 3.74μs 2.01ns 7.78ns 0.0262 0 0 1.9 KB
#4792 SendAsync netcoreapp3.1 4.47μs 2.28ns 8.52ns 0.0315 0 0 2.43 KB
#4792 SendAsync net472 7.12μs 2.56ns 9.25ns 0.473 0 0 2.99 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.39μs 0.506ns 1.82ns 0.0222 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.04μs 0.837ns 3.13ns 0.021 0 0 1.57 KB
master EnrichedLog net472 2.3μs 3.22ns 12.5ns 0.236 0 0 1.49 KB
#4792 EnrichedLog net6.0 1.31μs 0.843ns 3.04ns 0.0216 0 0 1.57 KB
#4792 EnrichedLog netcoreapp3.1 2.01μs 1.31ns 4.9ns 0.0212 0 0 1.57 KB
#4792 EnrichedLog net472 2.24μs 3.55ns 13.8ns 0.237 0 0 1.49 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 97.6ns 378ns 0.056 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 119μs 100ns 374ns 0 0 0 4.21 KB
master EnrichedLog net472 147μs 116ns 432ns 0.665 0.222 0 4.38 KB
#4792 EnrichedLog net6.0 112μs 81.5ns 316ns 0.056 0 0 4.21 KB
#4792 EnrichedLog netcoreapp3.1 117μs 95.1ns 356ns 0.059 0 0 4.21 KB
#4792 EnrichedLog net472 146μs 43.3ns 168ns 0.658 0.219 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.78μs 0.963ns 3.6ns 0.0295 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 3.85μs 1.37ns 5.29ns 0.0288 0 0 2.13 KB
master EnrichedLog net472 4.6μs 11.6ns 45.1ns 0.306 0 0 1.93 KB
#4792 EnrichedLog net6.0 3μs 1.02ns 3.67ns 0.0288 0 0 2.13 KB
#4792 EnrichedLog netcoreapp3.1 3.85μs 1.3ns 4.69ns 0.0287 0 0 2.13 KB
#4792 EnrichedLog net472 4.61μs 1.46ns 5.47ns 0.307 0 0 1.93 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 0.544ns 2.11ns 0.0155 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.59μs 0.596ns 2.07ns 0.0143 0 0 1.1 KB
master SendReceive net472 2.03μs 4.28ns 16.6ns 0.177 0 0 1.12 KB
#4792 SendReceive net6.0 1.32μs 0.963ns 3.47ns 0.0151 0 0 1.1 KB
#4792 SendReceive netcoreapp3.1 1.57μs 0.455ns 1.7ns 0.0149 0 0 1.1 KB
#4792 SendReceive net472 1.89μs 2.45ns 9.48ns 0.176 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.66μs 0.861ns 3.33ns 0.0214 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.61μs 2.2ns 8.53ns 0.0216 0 0 1.58 KB
master EnrichedLog net472 3.93μs 1.35ns 5.22ns 0.311 0 0 1.96 KB
#4792 EnrichedLog net6.0 2.6μs 1.34ns 5ns 0.021 0 0 1.53 KB
#4792 EnrichedLog netcoreapp3.1 3.64μs 1.81ns 7.01ns 0.0202 0 0 1.58 KB
#4792 EnrichedLog net472 4.06μs 1.14ns 4.42ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4792

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 1.165 812.50 946.19
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.160 407.87 473.01

Faster 🎉 in #4792

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.244 740.35 594.93

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 408ns 0.182ns 0.705ns 0.00751 0 0 536 B
master StartFinishSpan netcoreapp3.1 560ns 0.102ns 0.38ns 0.00729 0 0 536 B
master StartFinishSpan net472 740ns 0.975ns 3.78ns 0.0852 0 0 538 B
master StartFinishScope net6.0 529ns 0.147ns 0.568ns 0.00915 0 0 656 B
master StartFinishScope netcoreapp3.1 700ns 0.16ns 0.599ns 0.00883 0 0 656 B
master StartFinishScope net472 812ns 0.383ns 1.43ns 0.0979 0 0 618 B
#4792 StartFinishSpan net6.0 473ns 0.128ns 0.479ns 0.00738 0 0 536 B
#4792 StartFinishSpan netcoreapp3.1 561ns 0.17ns 0.66ns 0.00727 0 0 536 B
#4792 StartFinishSpan net472 595ns 0.137ns 0.529ns 0.0852 0 0 538 B
#4792 StartFinishScope net6.0 525ns 0.12ns 0.464ns 0.00908 0 0 656 B
#4792 StartFinishScope netcoreapp3.1 735ns 0.203ns 0.787ns 0.00896 0 0 656 B
#4792 StartFinishScope net472 946ns 0.86ns 3.33ns 0.0978 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 621ns 0.239ns 0.925ns 0.00928 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 898ns 3.25ns 12.6ns 0.00878 0 0 656 B
master RunOnMethodBegin net472 1.04μs 0.486ns 1.88ns 0.098 0 0 618 B
#4792 RunOnMethodBegin net6.0 613ns 0.2ns 0.776ns 0.00915 0 0 656 B
#4792 RunOnMethodBegin netcoreapp3.1 830ns 0.887ns 3.43ns 0.00859 0 0 656 B
#4792 RunOnMethodBegin net472 1.1μs 0.359ns 1.34ns 0.098 0 0 618 B

@andrewlock
Copy link
Member

andrewlock commented Oct 31, 2023 •
edited
Loading

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4792) (11.048M)   : 0, 11048163
    master (11.650M)   : 0, 11649912
    benchmarks/2.38.0 (11.867M)   : 0, 11866956
    benchmarks/2.9.0 (11.705M)   : 0, 11705393

    section Automatic
    This PR (4792) (7.655M)   : 0, 7654986
    master (7.936M)   : 0, 7935694
    benchmarks/2.38.0 (8.176M)   : 0, 8175636
    benchmarks/2.9.0 (8.428M)   : 0, 8428352

    section Trace stats
    This PR (4792) (7.995M)   : 0, 7994700
    master (8.167M)   : 0, 8166850
    benchmarks/2.38.0 (8.450M)   : 0, 8450370

    section Manual
    This PR (4792) (9.721M)   : 0, 9720802
    master (9.994M)   : 0, 9994262
    benchmarks/2.38.0 (10.334M)   : 0, 10334368

    section Manual + Automatic
    This PR (4792) (7.332M)   : 0, 7332443
    master (7.556M)   : 0, 7556155
    benchmarks/2.38.0 (7.750M)   : 0, 7750484

    section Version Conflict
    This PR (4792) (6.882M)   : 0, 6881614
    master (6.839M)   : 0, 6838643
    benchmarks/2.38.0 (7.137M)   : 0, 7136691
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4792) (9.748M)   : 0, 9748431
    master (9.544M)   : 0, 9543559
    benchmarks/2.38.0 (9.548M)   : 0, 9548121
    benchmarks/2.9.0 (9.418M)   : 0, 9417895

    section Automatic
    This PR (4792) (6.569M)   : 0, 6568815
    master (6.702M)   : 0, 6701712
    benchmarks/2.38.0 (6.747M)   : 0, 6747227

    section Trace stats
    This PR (4792) (6.903M)   : 0, 6902698
    master (6.863M)   : 0, 6863119
    benchmarks/2.38.0 (6.815M)   : 0, 6814846

    section Manual
    This PR (4792) (8.423M)   : 0, 8422995
    master (8.398M)   : 0, 8398442
    benchmarks/2.38.0 (8.263M)   : 0, 8263131

    section Manual + Automatic
    This PR (4792) (6.225M)   : 0, 6224692
    master (6.199M)   : 0, 6198506
    benchmarks/2.38.0 (6.275M)   : 0, 6275411

    section Version Conflict
    This PR (4792) (5.902M)   : 0, 5901512
    master (5.829M)   : 0, 5828745
    benchmarks/2.38.0 (5.670M)   : 0, 5669744
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4792) (9.470M)   : 0, 9469975
    master (9.530M)   : 0, 9529750
    benchmarks/2.38.0 (9.007M)   : 0, 9007066
    benchmarks/2.9.0 (9.752M)   : 0, 9752446

    section Automatic
    This PR (4792) (6.673M)   : 0, 6673212
    master (6.622M)   : 0, 6622063
    benchmarks/2.38.0 (6.423M)   : 0, 6422756
    benchmarks/2.9.0 (6.987M)   : 0, 6986655

    section Trace stats
    This PR (4792) (7.034M)   : 0, 7033611
    master (6.910M)   : 0, 6910193
    benchmarks/2.38.0 (6.721M)   : 0, 6720838

    section Manual
    This PR (4792) (8.353M)   : 0, 8352724
    master (8.223M)   : 0, 8222912
    benchmarks/2.38.0 (7.977M)   : 0, 7977494

    section Manual + Automatic
    This PR (4792) (6.512M)   : 0, 6511793
    master (6.401M)   : 0, 6401260
    benchmarks/2.38.0 (6.349M)   : 0, 6349066

    section Version Conflict
    This PR (4792) (5.828M)   : 0, 5828058
    master (5.831M)   : 0, 5831118
    benchmarks/2.38.0 (5.682M)   : 0, 5681654
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.578M)   : 0, 7577773
    benchmarks/2.38.0 (7.578M)   : 0, 7577544
    benchmarks/2.9.0 (8.053M)   : 0, 8052960

    section No attack
    master (2.178M)   : 0, 2178299
    benchmarks/2.38.0 (2.184M)   : 0, 2184279
    benchmarks/2.9.0 (3.237M)   : 0, 3237284

    section Attack
    master (1.724M)   : 0, 1723721
    benchmarks/2.38.0 (1.711M)   : 0, 1710542
    benchmarks/2.9.0 (2.483M)   : 0, 2482504

    section Blocking
    master (3.438M)   : 0, 3437779
    benchmarks/2.38.0 (3.486M)   : 0, 3486124

    section IAST default
    master (6.813M)   : 0, 6813043

    section IAST full
    master (6.249M)   : 0, 6248788

    section Base vuln
    master (0.924M)   : 0, 924395

    section IAST vuln
    master (0.908M)   : 0, 908012
Loading

@andrewlock andrewlock force-pushed the bot/test-package-versions-bump branch from 9c24e29 to 80b416b Compare November 1, 2023 15:29
@github-actions github-actions bot force-pushed the bot/test-package-versions-bump branch 2 times, most recently from 2d2dcb3 to 4823856 Compare November 1, 2023 16:19
@andrewlock andrewlock force-pushed the bot/test-package-versions-bump branch from 4823856 to 9708134 Compare November 1, 2023 16:29
andrewlock
andrewlock reviewed Nov 1, 2023
View reviewed changes
tracer/build/PackageVersionsGeneratorDefinitions.json Outdated
@@ -38,7 +38,7 @@
"MinVersion": "1.4.0",
"MaxVersionExclusive": "2.0.0",
"SpecificVersions": [
"1.8.*"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spotted that we were pinning to the wrong version here so made it float properly instead

@andrewlock andrewlock force-pushed the bot/test-package-versions-bump branch from 9708134 to 5fe464c Compare November 2, 2023 09:26
@andrewlock andrewlock force-pushed the bot/test-package-versions-bump branch from 5fe464c to 56a5b56 Compare November 2, 2023 09:31
andrewlock
andrewlock reviewed Nov 2, 2023
View reviewed changes
tracer/dependabot/Datadog.Dependabot.Integrations.csproj
Comment on lines -241 to 242
<!-- Latest package https://www.nuget.org/packages/xunit/2.5.3 -->
<!-- Latest package https://www.nuget.org/packages/xunit/2.6.0 -->
<PackageReference Include="xunit" Version="2.5.3" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They haven't released a 2.6.0 of xunit.extensibility.vstudio yet, so bumping this breaks CI. Hopefully they release it soon otherwise we'll need to take another look at the tests..

andrewlock
andrewlock approved these changes Nov 2, 2023
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewlock andrewlock merged commit 6626ee8 into master Nov 2, 2023
@andrewlock andrewlock deleted the bot/test-package-versions-bump branch November 2, 2023 14:34
@github-actions github-actions bot added this to the vNext milestone Nov 2, 2023
@andrewlock andrewlock added area:tests unit tests, integration tests area:test-apps apps used to test integrations area:dependabot dependabot updates labels Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock approved these changes

@kevingosse kevingosse kevingosse approved these changes

Assignees
No one assigned
Labels
area:dependabot dependabot updates area:test-apps apps used to test integrations area:tests unit tests, integration tests
Projects
None yet
Milestone
2.41.0
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewlock @kevingosse